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MEASURING TEACHERS' MATHEMATICAL KNOWLEDGE 



Margaret Heritage and Terry Vendlinski 
CSE/CRESST, UCLA 

Abstract 

Teachers' knowledge of mathematics is pivotal to their capacity to provide effective 
mathematics instruction and to their ability to assess student learning (Ball, Hill, & 

Bass, 2005; Ma, 1999; Schifter, 1999). The National Council for the Teaching of 
Mathematics (NCTM, 2000) makes it clear that teachers need knowledge of the 
whole domain as well as knowledge about the important ideas that are central to 
their grade level. POWERSOURCE is expected, through professional development 
and job aids, to influence teachers' pedagogical content knowledge and assessment 
practices. To gauge such effects we have developed teacher measures that focus on 
three key mathematical principles that are central to POWERSOURCE: the 
distributive property, solving equations, and rational number equivalence. 

Background 

The measurement of teachers' knowledge of mathematics has been a problem 
occupying researchers for several decades. Early efforts, consonant with the assumption 
that knowledge of and skill with mathematics content is critical to teaching, used 
characteristics of teachers and their educational background; for example, the number 
of mathematics courses taken and degrees obtained (Begle, 1979). From his meta- 
analysis of studies conducted between 1960 and 1976 that examined the effects of 
teacher characteristics on student performance Begle concluded that the idea that the 
more subject knowledge a teacher had, the better the teacher, needed 'drastic 
modification' (p.51). 

Later studies, influenced by the notion of pedagogical content knowledge 
(Shulman, 1986, 1987; Wilson, Shulman & Rickert, 1987) focused on qualitative probing 
of forms of mathematical knowledge that were more closely tied to teaching (e.g.. Ball, 
1991; Kennedy, 1997; Ma, 1999). A further approach, developed from this prior work. 
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focuses on mathematics knowledge for teaching (Ball & Bass, 2000; Hill, Rowan & Ball, 
2005). Mathematics knowledge for teaching refers to the knowledge that is specific to 
the profession of teaching (as opposed to the kind of knowledge used by other 
professions such as accounting and engineering) and is closely linked to student 
achievement. For example. Ball (1990) describes the difference between a teacher's 
ability to execute an operation such as division by fractions and the kind of 
understanding needed for teaching, namely, how to explain what the operation means 
and why it works. 

Researchers from the Study of Instructional Improvement (SII) designed multiple- 
choice survey instruments to measure growth in teachers' mathematical knowledge 
used in elementary school mathematics and to provide more information about the 
ways in which teacher knowledge contributes to student achievement. These 
instruments were used in a study of the effects of teacher knowledge for teaching on 
student achievement and the results showed that teachers who scored higher on these 
measures of mathematics knowledge for teaching produced better gains on student 
achievement (Ball, Hill & Bass, 2005; Hill, Rowan, &Ball, 2005). Recently, the same 
researchers have developed similar instruments to measure teachers' knowledge of pre- 
algebra and algebra. 

Because of the scarcity of valid and reliable instruments to gauge teachers' 
mathematical knowledge for teaching, to date, it has been difficult to measure the 
outcomes of professional development programs designed to improve teachers' 
knowledge. Recently, Hill and Ball, 2004 used the SII measures to evaluate the outcomes 
of California's Mathematics Professional Development Institutes and were able to show 
that 'teachers can learn mathematics for elementary school teaching in the context of a 
single professional development program' (p.345). 

This paper will present further conceptualizations of measuring sixth- , seventh- , 
and eighth-grade teachers' mathematical knowledge, describe the development of the 
measures and a validation study of the measures and conclude with possibilities for 
using this conceptualization in other contexts. 

Further Conceptualizations of Instruments to Measure Teacher Knowledge 

The conceptualization of instruments to measure teacher knowledge presented in 
this paper builds on the prior research from the SII. We have developed and are 
validating two instruments designed to gauge teachers' mathematical knowledge to 
teach for understanding key principles underlying mastery of Algebra I, specifically, the 
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distributive property, solving equations and rational number equivalence. The two 
instruments conceptualize the measurement of teacher knowledge as knowledge that is 
central to and embedded in the every day practice of teaching, irrespective of teachers' 
specific curriculum or approach to teaching. This would include the knowledge that 
teachers draw on to interpret students' understanding of mathematical ideas and plan 
instruction (Ball, Lubienski & Mewborn, 2001; NRC, 2001a; Shulman, 1986, 1987) to give 
students feedback (NRC, 2000, 2001b) and to explain, justify and model mathematical 
ideas to students (NRC, 2001b). 

One instrument is a series of performance tasks in which teachers are asked to 
review student responses to assessments and to respond to a series of questions. Figure 
1 shows a student's response to an assessment addressing understanding of the 
distributive property. 
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In problems 3-4, you will took at another student's work and try to Nl to 
any missing pieces. To show you how to do this, here's an example with 
NO missing pieces: 
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Mere are two problems that are different from the example. Fid in the 
missing pieces: 



| A student simplified ibr expression 4<5 ♦ 2) m 4 steps. Can you fill m the ir 
numbers in steps 2. 3 and 4? 
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Figure 1. Student response to an assessment checking understanding of the distributive 
property 
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After the teachers have reviewed the student work, they are asked to respond to 
the following questions: 

1. What is the key principle that these assessments address? Why do students need 
to understand this principle for Algebra I? 

2. What inferences would you draw from this student's responses? What does this 
student know? What does this student not know? 

3. If you were this child's teacher what written feedback would you give to this 
student? 

4. If this student were in your class, based on your responses to questions 2 and 3, 
what would you do next in your instruction? 

Also included are some tasks that require the teacher to describe how they would 
teach certain principles, (e.g., how would you explain the distributive property to 
students?) 

The second instrument is a survey that asks teachers to rate their level of expertise, 
on a five-point scale ranging from novice to expert, in teaching key principles for 
understanding. For example, rating their expertise in 'showing and justifying how 
defining multiplication as repeated addition can be used to multiply fractions by 
integers' and in 'explaining how to use the distributive property to add fractions with 
the same denominator.' 

Prior CRESST research found that teachers' own survey ratings of their subject 
knowledge were related to student achievement (Boscardin et al, 2005) and suggests 
that self-rating could be a reliable way of measuring teacher knowledge. 

Developing the measures 

The foundation for both measures is a map of the domain of Algebra I, developed 
at CRESST in collaboration with algebraists from the University of Chicago. The map 
represents the key principles of Algebra I, and the relationship between and amongst 
them. The key principles have not been derived from any particular curricula, but they 
are reflected in a range of the state's mathematics standards and in the NCTM 
standards. 

To design the survey, a group of expert mathematics teachers and mathematicians 
analyzed the three key principles and several curricula to determine what was involved 
in teaching the key principles for understanding. Survey items for each key principle 
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were developed from this analysis to reflect the actual knowledge that teachers would 
need to teach for student understanding. 

The performance tasks were developed from a pilot study of formative 
assessments to assess students' understanding of key principles. A group of 
mathematicians and expert mathematics teachers analyzed student responses and 
selected those that showed gaps in knowledge, misconceptions, and understanding of 
the key principles. The group then designed tasks for teachers to complete that were 
related to the student responses and that required teachers to draw on their 
mathematical knowledge for teaching the principles for understanding. For example, 
from the student's response to an assessment of rational number equivalence the 
teachers were asked, what do you think this student understands about rational 
number equivalence? What does the student not understand? What would you do next 
in your instruction? 

A group of eight mathematics teachers of varying levels of experience and 
expertise completed the performance tasks, and to capture the range of knowledge 
represented four point scoring rubrics were derived from an analysis of their written 
responses. For example, in response to the question about the principle addressed by 
the assessment shown in Figure 1 and why it is essential for algebra, responses ranged 
from 'order of operations - essential for solving problems' to 'distributive property, 
which is needed in Algebra I to solve algebraic equations and multiply monomials and 
polynomials.' 

The rubric for the task of explaining what the teachers' next instructional steps 
would be is shown in Figure 2. This rubric assesses the degree to which the teacher 
responses show detailed and principle-based knowledge of distribution. A score point 
of 4 shows that the teacher understands the distributive property as repeated addition, 
factoring as distribution in reverse, using the distributive property with whole 
numbers, and generalizing the distributive property to other numbers and variables; 
score point of 3 shows that the teacher has an understanding of either the distributive 
property as repeated addition or of factoring as distribution in reverse and of using the 
distributive property with at least whole numbers; a score point of 2 shows the teacher 
has a rudimentary, procedural, rather than principle-based understanding of the 
properties of arithmetic; and a score point of 1 indicates that the teacher has no 
knowledge or understanding of the distributive property. 
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• Explain the distributive property as repeated 
addition 

• Explain factoring as distribution in reverse 

• Model the use of the distributive property with 
whole numbers 

• Model generalizing to other numbers and 
variables 
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Either 

• Explain the distributive property as repeated 
addition 

Or 

• Explain factoring as distribution in reverse 

• Model the use of the distributive property with at 
least whole numbers 


2 


• Explain procedures for how to use the distributive 
property, equating procedures with the order of 
operations 


1 


• No explanation of the distributive property 



Figure 2. Scoring rubric for teaching the distributive property. 



Validation Study 

Currently, a study is being conducted to validate the self-rating survey and the 
performance tasks. One hundred, sixth-grade teachers have been recruited to 
participate in the study and they will complete the survey and performance tasks, and a 
multiple-choice measure of mathematics knowledge for pre-algebra and algebra, 
developed by researchers from the SII that address the key principles. The multiple- 
choice items will be scored according to the answer key for the measure, and after rater 
training, retired mathematics teachers will score the performance assessments using the 
rubrics for each task. 
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Reliability of scoring and the consistency of the responses to the various measures 
will be examined with generalizability studies. Factor analysis and correlation will be 
used to examine the distinctness and/ or generality of the various measures. Results will 
be used to provide validity evidence for teacher measures used in the POWERSOURCE 
study and to define the most efficient set of measuring changes in teacher knowledge. 

Conclusion 

Currently, few valid and reliable measures exist to assess outcomes of professional 
development. Until recently, measures of teacher knowledge, such as licensing tests for 
example, focused on computational knowledge rather than on the kind of mathematical 
knowledge that teachers need to use in the context of the classroom. Therefore, 
professional development programs designed to improve teachers' knowledge for 
teaching lacked the appropriate instruments to measure growth. 

The measures described in this paper have been designed to gauge growth in 
teachers' mathematical knowledge for teaching three key principles underlying mastery 
in algebra. However, we foresee that the conceptualization of measuring teacher 
knowledge that we have presented has the potential to adapt to other areas of 
mathematics and other subject areas, and to be used to evaluate teacher professional 
development programs. 
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